Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BlockifyVR (Virtual Reality) #1732

Draft
wants to merge 15 commits into
base: master
Choose a base branch
from

Conversation

Brackets-Coder
Copy link

@Brackets-Coder Brackets-Coder commented Oct 23, 2024

This is a very-much-still-work-in-progress Virtual Reality extension I've been working on since January of 2023.

The end goal is to have:

  • Little to no performance overhead with the extension.
  • Cross-platform compatibility for major 6DOF headsets supported by A-frame (Oculus, Vive, and Windows Mixed reality)
  • An easy to use system that's coherent and efficient
  • Works well with any 3D engine

Right now, it meets most of these standards, but there's always room for improvement.

Here are some things to note:

  • It includes a local copy of the A-frame JS library, version 1.6.0, MIT License. This is for VR Management, rendering, and ease of development as I'd have to write the entire thing from scratch using WebXR, WebGL, and the Gamepad API otherwise
  • Gallery thumbnail, documentation, and sample projects are not currently available and will be worked on as I get nearer towards release
  • The rendering uses a plane fixed to the camera which directly mirrors the scratch stage with an unshaded texture each frame. This is relatively performant and works well, but there could be better solutions to look into. The plane is placed 1 meter away from the camera and A-frame uses stereoscopic rendering to reduce eye strain. Note that the plane automatically scales to the size of the camera (display) while attempting to maintain the ratio of the stage. I use this instead of drawing directly to the canvas because it won't let me and I would have to re-write the renderer that's already tailored for VR.
  • Sorry the code's a bit messy

With all of that said, here's a quick (and very outdated) video of the first somewhat working test of the extension:

Untitled3.mp4

Please note that this video is an example from a very early version of the extension that only partially supported headset rotational tracking. The latest development version now supports 6DOF for both the headset and controllers and has button mapping and detection for cross platform controllers.

Upcoming release plan:

v1.0-pre-alpha: Earliest version, showcased in the video above. Minimal functionality, proof-of-concept version. Not available here. Not open source.

v1.0-alpha: work-in-progress. Expect significant bugs and poor performance. Should not be used.

v1.0-beta: Current version. Cross-platform compatibility, major bug fixes, and optimizations added. Working on optimizations. May last several weeks depending on severity of issues.

v1.0-release-candidate: Feature complete. Meets standards of full release with minimal issues, includes preparations for full release. This means gallery thumbnail, sample project, documentation, etc. Should last about a week.

v1.0: First release. Feature complete and meets all standards of great performance, very few issues, cross-platform compatibility, and intuitive design.

v1.5: Bugfixes and optimizations. Possibly a small feature or change, such as controller vibrations.

v2.0: Big update. Most likely something like hand tracking support for some platforms. Also will include some minor bugfixes and optimizations.

v2.2: Bugfixes and optimizations.

Active Development

v1.0-beta

  • Bug fixes
  • Optimizations
  • Minor block updates and Improvements

@Brackets-Coder Brackets-Coder marked this pull request as ready for review October 24, 2024 19:02
@Brackets-Coder Brackets-Coder marked this pull request as draft October 24, 2024 19:02
@Brackets-Coder Brackets-Coder changed the title BlockifyVR (Virtual Reality) Extension New Extension: BlockifyVR (Virtual Reality) Oct 24, 2024
@Brackets-Coder Brackets-Coder changed the title New Extension: BlockifyVR (Virtual Reality) BlockifyVR (Virtual Reality) Oct 24, 2024
@SharkPool-SP SharkPool-SP added the pr: new extension Pull requests that add a new extension label Oct 24, 2024
@hammouda101010
Copy link

that looks cool

@Brackets-Coder
Copy link
Author

that looks cool

It looks cool on the flat screen right now, but it's not so great when you try it. You'd probably get motion sickness. I'm still working on it.

@hammouda101010
Copy link

that looks cool

It looks cool on the flat screen right now, but it's not so great when you try it. You'd probably get motion sickness. I'm still working on it.

sadly i dont think that's possible to stop this.

@Brackets-Coder

This comment was marked as off-topic.

@Xeltalliv
Copy link
Contributor

Does this show the same scratch 3D perspective to both eyes?

@Brackets-Coder
Copy link
Author

Brackets-Coder commented Oct 29, 2024

Does this show the same scratch 3D perspective to both eyes?

A-frame has stereoscopic rendering built in and I've enabled it in the rendering settings, so no. This fixes some issues with visualizing depth, so the eye offset allows your brain to process the image with more clarity. Even if the matrices aren't used in the renderer, it should properly use the correct eye offset. Why do you ask, just out of curiosity? If you'd like I could add a block that disables or enables this feature, in case the Scratcher wants to build that framework themselves, which I'd doubt.

Keep in mind this is still a work in progress and most of the code is many months old so it's not the most efficient way to do VR.

@Xeltalliv
Copy link
Contributor

Why do you ask, just out of curiosity?

Yes. At one point I tried to extend my AR extension to support AR headsets and VR as well, but the multiple eye rendering has been the main thing holding me back. I was just curious to know how you solved it.

@Brackets-Coder
Copy link
Author

Brackets-Coder commented Oct 29, 2024

Why do you ask, just out of curiosity?

Yes. At one point I tried to extend my AR extension to support AR headsets and VR as well, but the multiple eye rendering has been the main thing holding me back. I was just curious to know how you solved it.

I believe that there can be two XRViewerPose matrices for each eye if available, according to Mozilla Docs. You might want to look into that again. It'd be great if Augmented Reality could support AR headsets such as the Oculus Quest 3 in addition to just mobile touchscreen devices. There should be a views array that can do this. An example can be seen here. Just a suggestion though. I simply think having AR headsets would be a nice update.

@Brackets-Coder
Copy link
Author

Brackets-Coder commented Oct 29, 2024

@Xeltalliv Your extensions are good quality and you seem to know what you're doing with WebXR.

I've looked at the code like 100 times. Do you mind taking a peek at the left-controller-manager and right-controller-manager A-frame components? Pressing any button significantly drops the FPS even without calling the event hat block. Additionally, pressing any button on the right controller returns its left controller counterparts except for X, Y, A, and B on Oculus Quest.

Just asking for another pair of eyes, that's all. I've asked other developers I know and I've asked ChatGPT and I've referenced the A-frame docs and everything looks fine to me, I just can't narrow down the issue.

@Xeltalliv
Copy link
Contributor

Xeltalliv commented Oct 30, 2024

I'll take a look when I have time. Thought it will probably be hard because the only kind of VR I have access to is phone-based 3DOF no controller VR.

Also, do you mind giving some links, so that I can faster figure out where to look?

@Brackets-Coder
Copy link
Author

I'll take a look when I have time. Thought it will probably be hard because the only kinf of VR I have access to is phone-based 3DOF no controller VR.

Also, do you mind giving some links, so that I can faster figure out where to look?

Ok, no problem! I wasn't asking you to test the extension, and Google Cardboard-like devices aren't supported by it anyway, so you're fine, don't worry about it.

As for the links, do you mean the XRViewerPose Mozilla Developer Docs links or the links to my controller code? Because as of right now the latest version of my extension is only available on private GitHub repo accessible only to me. You could always just view the added BlockifyVR.js commit here and scroll down to the [hand]-controller-component blocks of code. Just a note though, it does include the entire minified A-frame library, so it might lag your browser a bit.

@Brackets-Coder
Copy link
Author

I'll take a look when I have time. Thought it will probably be hard because the only kinf of VR I have access to is phone-based 3DOF no controller VR.

Also, do you mind giving some links, so that I can faster figure out where to look?

As for VR testing, you wouldn't really need a VR headset. I've used Meta Quest's Immersive Web Emulator extension for testing when I didn't care to load up the entire headset.

@Drago-Cuven
Copy link

ah yes, I remember seeing a trailer for this on scratch. I wanted to help, but now I can't

@Drago-Cuven
Copy link

nvm I think I can
i'm Martinelplayz on scratch btw

@Brackets-Coder

This comment was marked as resolved.

@siskka7

This comment was marked as resolved.

@Thebloxers998
Copy link

could i try the extension?

Hi siskka7 it's me Thebloxers998

@Brackets-Coder

This comment was marked as resolved.

@siskka7
Copy link

siskka7 commented Nov 9, 2024

Wait is the website released?

@Drago-Cuven
Copy link

Meta quest 2 have thumb rest place detecting could be a good update. touching and pressing is something different there could be block touching?

I already made that suggestion

@Brackets-Coder
Copy link
Author

(Blocks ideas) bending screen so you can see everything on screen

Not sure what you mean by this.

Meta quest 2 have thumb rest place detecting could be a good update. touching and pressing is something different there could be block touching?

Yes, it's quite possible that there will be some blocks for button touch detection along with button press detection. However, I'm still working on this.

Wait is the website released?

No, and most likely it won't ever be released.

@siskka7
Copy link

siskka7 commented Nov 11, 2024

i know it must work with any 3d engine but in https://studio.penguinmod.com/editor.html?livetests in extension tab is 3d and vr extension and it is very wobbly extension. i think it works similar to your extension

@Brackets-Coder
Copy link
Author

i know it must work with any 3d engine but in https://studio.penguinmod.com/editor.html?livetests in extension tab is 3d and vr extension and it is very wobbly extension. i think it works similar to your extension

Yes, I'm aware there are other VR extensions out there, but last time I checked the one for Penguin Mod was only available with their 3D extension, which is something I'm trying to avoid. Additionally, this extension is targeted towards Turbowarp and aims to have little to know performance overhead, as well as being intuitive and easy to use.

@siskka7
Copy link

siskka7 commented Nov 11, 2024

i know it must work with any 3d engine but in https://studio.penguinmod.com/editor.html?livetests in extension tab is 3d and vr extension and it is very wobbly extension. i think it works similar to your extension

Yes, I'm aware there are other VR extensions out there, but last time I checked the one for Penguin Mod was only available with their 3D extension, which is something I'm trying to avoid. Additionally, this extension is targeted towards Turbowarp and aims to have little to know performance overhead, as well as being intuitive and easy to use.

FUN FACT:i dont even know how to make 3d projects D:

@Thebloxers998
Copy link

i know it must work with any 3d engine but in https://studio.penguinmod.com/editor.html?livetests in extension tab is 3d and vr extension and it is very wobbly extension. i think it works similar to your extension

Yes, I'm aware there are other VR extensions out there, but last time I checked the one for Penguin Mod was only available with their 3D extension, which is something I'm trying to avoid. Additionally, this extension is targeted towards Turbowarp and aims to have little to know performance overhead, as well as being intuitive and easy to use.

FUN FACT:i dont even know how to make 3d projects D:

Same 💀

@Brackets-Coder
Copy link
Author

Brackets-Coder commented Nov 11, 2024

i know it must work with any 3d engine but in https://studio.penguinmod.com/editor.html?livetests in extension tab is 3d and vr extension and it is very wobbly extension. i think it works similar to your extension

Yes, I'm aware there are other VR extensions out there, but last time I checked the one for Penguin Mod was only available with their 3D extension, which is something I'm trying to avoid. Additionally, this extension is targeted towards Turbowarp and aims to have little to know performance overhead, as well as being intuitive and easy to use.

FUN FACT:i dont even know how to make 3d projects D:

You can download this .zip file, rename the file extension to .sb3, and import it into Turbowarp how you would normally import a .sb3 file. It's a demo project hosting an already-set-up-for-you Simple3D scene with basic VR integration.

You guys should be able to use this for testing. Hope this helps.

Simple3D Engine VR.zip

@Thebloxers998
Copy link

i know it must work with any 3d engine but in https://studio.penguinmod.com/editor.html?livetests in extension tab is 3d and vr extension and it is very wobbly extension. i think it works similar to your extension

Yes, I'm aware there are other VR extensions out there, but last time I checked the one for Penguin Mod was only available with their 3D extension, which is something I'm trying to avoid. Additionally, this extension is targeted towards Turbowarp and aims to have little to know performance overhead, as well as being intuitive and easy to use.

FUN FACT:i dont even know how to make 3d projects D:

You can download this .zip file, rename the file extension to .sb3, and import it into Turbowarp how you would normally import a .sb3 file. It's a demo project hosting an already-set-up-for-you Simple3D scene with basic VR integration.

You guys should be able to use this for testing. Hope this helps.

Simple3D Engine VR.zip

Nice, I'm going to try it

@Thebloxers998
Copy link

Imagine someone makes a video game with it, it would be super cool

@Brackets-Coder
Copy link
Author

i know it must work with any 3d engine but in https://studio.penguinmod.com/editor.html?livetests in extension tab is 3d and vr extension and it is very wobbly extension. i think it works similar to your extension

Yes, I'm aware there are other VR extensions out there, but last time I checked the one for Penguin Mod was only available with their 3D extension, which is something I'm trying to avoid. Additionally, this extension is targeted towards Turbowarp and aims to have little to know performance overhead, as well as being intuitive and easy to use.

FUN FACT:i dont even know how to make 3d projects D:

You can download this .zip file, rename the file extension to .sb3, and import it into Turbowarp how you would normally import a .sb3 file. It's a demo project hosting an already-set-up-for-you Simple3D scene with basic VR integration.
You guys should be able to use this for testing. Hope this helps.
Simple3D Engine VR.zip

Nice, I'm going to try it

Note that even in beta with the significant bug fixes and optimizations I've made, performance still depends on scene complexity (and project optimization) as well as hardware specifications. Also, there are a few bugs still to be fixed and there is no button touch detection yet - pressing them still works though.

@Thebloxers998
Copy link

By the way, in the project, what are some things to mess around with

@Brackets-Coder
Copy link
Author

Brackets-Coder commented Nov 11, 2024

Imagine someone makes a video game with it, it would be super cool

I've thought about possibly making a simple Beatsaber demo using my extension and Simple3D, but it would only consist of a single one-minute-long song and would have simple graphics complexity for performance reasons.

The problem is that I don't really know how to do three-dimensional collision detection yet and that's quite essential for almost everything VR related; there's just so much interaction that needs to be integrated, so it's probably not going to happen.

@Brackets-Coder
Copy link
Author

By the way, in the project, what are some things to mess around with

I'm not sure what you mean. You could try adding some meshes and things, but right now the only functionality in the demo project I've provided is basic headset and controller tracking with a voice that speaks the most recently pressed button. That's about it. You could try walking around and seeing what happens when you press a button.

@Thebloxers998
Copy link

By the way, in the project, what are some things to mess around with

I'm not sure what you mean. You could try adding some meshes and things, but right now the only functionality in the demo project I've provided is basic headset and controller tracking with a voice that speaks the most recently pressed button. That's about it. You could try walking around and seeing what happens when you press a button.

I mean like settings that you change basically

@Brackets-Coder
Copy link
Author

By the way, in the project, what are some things to mess around with

I'm not sure what you mean. You could try adding some meshes and things, but right now the only functionality in the demo project I've provided is basic headset and controller tracking with a voice that speaks the most recently pressed button. That's about it. You could try walking around and seeing what happens when you press a button.

I mean like settings that you change basically

There's no settings to be changed right now except for "movement speed" which is not functional at all; it only works when falling back to computer controls if there is a lack of a VR headset presence. Additionally, it doesn't take into account of Delta Time so it doesn't really work well when changing FPS.

@Thebloxers998
Copy link

By the way, in the project, what are some things to mess around with

I'm not sure what you mean. You could try adding some meshes and things, but right now the only functionality in the demo project I've provided is basic headset and controller tracking with a voice that speaks the most recently pressed button. That's about it. You could try walking around and seeing what happens when you press a button.

I mean like settings that you change basically

There's no settings to be changed right now except for "movement speed" which is not functional at all; it only works when falling back to computer controls if there is a lack of a VR headset presence. Additionally, it doesn't take into account of Delta Time so it doesn't really work well when changing FPS.

Oh ok

@siskka7
Copy link

siskka7 commented Nov 15, 2024

Hmm

(Blocks ideas) bending screen so you can see everything on screen

Not sure what you mean by this.

Meta quest 2 have thumb rest place detecting could be a good update. touching and pressing is something different there could be block touching?

Yes, it's quite possible that there will be some blocks for button touch detection along with button press detection. However, I'm still working on this.

Wait is the website released?

No, and most likely it won't ever be released.

I mean the plane that mirrors stage will have option to bend like meta's Windows (screens)

@siskka7
Copy link

siskka7 commented Nov 15, 2024

And next idea more layers i mean like 3d (warning this idea is very stupid)

@Thebloxers998
Copy link

And next idea more layers i mean like 3d (warning this idea is very stupid)

I don't know if that should be added, what if it affects performance

@Brackets-Coder
Copy link
Author

And next idea more layers i mean like 3d (warning this idea is very stupid)

I don't know if that should be added, what if it affects performance

The problem with this is that the Scratch Stage is a 2D image, there's no depth information to be translated into the Immersive Web experience so that's why I use a Plane. All 3D rendering happens in the project. Stereoscopy is applied automatically to give the illusion of depth.

@siskka7
Copy link

siskka7 commented Nov 16, 2024

And next idea more layers i mean like 3d (warning this idea is very stupid)

I don't know if that should be added, what if it affects performance

The problem with this is that the Scratch Stage is a 2D image, there's no depth information to be translated into the Immersive Web experience so that's why I use a Plane. All 3D rendering happens in the project. Stereoscopy is applied automatically to give the illusion of depth.

i mean like a more planes but the plane must be somwere invisible (but this idea still is stupid )

@Brackets-Coder
Copy link
Author

And next idea more layers i mean like 3d (warning this idea is very stupid)

I don't know if that should be added, what if it affects performance

The problem with this is that the Scratch Stage is a 2D image, there's no depth information to be translated into the Immersive Web experience so that's why I use a Plane. All 3D rendering happens in the project. Stereoscopy is applied automatically to give the illusion of depth.

i mean like a more planes but the plane must be somwere invisible (but this idea still is stupid )

That still can't work because the stage is 2D, all of the 3D stuff is managed by the project and my extension has no knowledge or control over it.

@Thebloxers998
Copy link

And next idea more layers i mean like 3d (warning this idea is very stupid)

I don't know if that should be added, what if it affects performance

The problem with this is that the Scratch Stage is a 2D image, there's no depth information to be translated into the Immersive Web experience so that's why I use a Plane. All 3D rendering happens in the project. Stereoscopy is applied automatically to give the illusion of depth.

i mean like a more planes but the plane must be somwere invisible (but this idea still is stupid )

That still can't work because the stage is 2D, all of the 3D stuff is managed by the project and my extension has no knowledge or control over it.

True

@Thebloxers998
Copy link

By the way, is this compatible with the AR extension

@Brackets-Coder
Copy link
Author

Brackets-Coder commented Nov 16, 2024

By the way, is this compatible with the AR extension

No. They're two separate extensions.

@siskka7
Copy link

siskka7 commented Nov 17, 2024

And next idea more layers i mean like 3d (warning this idea is very stupid)

I don't know if that should be added, what if it affects performance

The problem with this is that the Scratch Stage is a 2D image, there's no depth information to be translated into the Immersive Web experience so that's why I use a Plane. All 3D rendering happens in the project. Stereoscopy is applied automatically to give the illusion of depth.

i mean like a more planes but the plane must be somwere invisible (but this idea still is stupid )

That still can't work because the stage is 2D, all of the 3D stuff is managed by the project and my extension has no knowledge or control over it.

Ok

@siskka7
Copy link

siskka7 commented Nov 17, 2024

Block ideas detecting floor y

@Drago-Cuven
Copy link

Block ideas detecting floor y

OH YEAH! it would be useful to be able to tell where the floor is "supposed" to be.

@Brackets-Coder
Copy link
Author

Block ideas detecting floor y

OH YEAH! it would be useful to be able to tell where the floor is "supposed" to be.

I disagree to be honest. Floor level is internally managed by A-frame, and while it would be possible to make such a block, it wouldn't be practical or serve as much as a significant purpose that the other blocks serve. Also, we're out of alpha and into beta so new features should be minuscule and important if they're going to be included at all. For now, I'm focusing on bug fixes and optimizations.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
pr: new extension Pull requests that add a new extension
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants